Learnings

Outcomes

Project outcomes are subdivided into two sections. The former looks at the findings gathered from stakeholders on the impact of different innovation project types on customers in the situations considered for this study. The latter focuses on the structure and functionalities of the tool through which users can access the findings gathered from stakeholders to assess the impact of innovation projects considered. 

• Findings gathered from stakeholders
  • We engaged stakeholders to gather their views on how customers in various situations (as defined at the onset of the project and based on the initial scope provided by the energy networks) would be impacted by different types of innovation projects. As previously mentioned, stakeholder feedback was captured through a bespoke online survey.
  • In the survey, each stakeholder was asked how a type of project (illustrated by a practical scenario) would impact customers in a specific situation. For example, stakeholders were asked to rate the impact of a project that ultimately reduces household costs (i.e. increases their disposable income) on customers who live in flats or houses or on customers who are owner-occupiers as opposed to private renters or social renters (and vice versa). Stakeholders were able to rate each impact from ‘high positive’ to ‘high negative’ on a 5-point scale.
  • Anonymised results of these surveys are available on request
• Structure and functionalities of the Consumer Vulnerability Assessment Tool: The tool is built on Microsoft Excel and is composed of four different tabs. This section describes the key elements and functionalities contained within each of these tabs.
  • ‘Guide’ tab – this tab is the first to appear once a user opens the file. It includes high level guidance on how to use the tool for the first time
  • ‘Questions’ tab – this tab is the main user interface of the tool:
    • Questions – The tab presents ten questions which the user must answer for the tool to produce an output. The questions that follow are used to assess whether the project would lead to a positive or negative impact from the customers’ perspective. If the project does not lead to either a positive or negative change from the customers’ perspective, the user can select the ‘No Change/Unrelated’ option. Users can only select one of the three options for each question and must answer all of them before proceeding.
    • Timeframe of impact – Users can choose to select an option from these dropdown boxes to indicate the timeframe over which the 'impacts' listed will be felt by customers. For example, a project may have various impacts on customers immediately (i.e. in the ‘short term’) but any increase/decrease in costs will only play out over the long term. Note that selecting an option from the dropdown boxes does not impact the scores of the project being assessed. The selection is presented within the detailed Impact Matrix.
    • Additional comments – Users can enter text in this box to explain the reasoning behind the answer given to the question. This is particularly useful for audit purposes. The explanations will be reported on the ‘Detailed Results’ tab.
    • ‘See Results’ button – This button takes the user to the ‘Results Dashboard’ tab which is the main output interface of the tool.
  • ‘Results Dashboard’ tab – This tab presents the user with the key outputs of the tool for the project being assessed:
    • Project Impact Score
      • Overall Project Score – This is the main output of the tool. The score, expressed out of 10, is a measure of how beneficial the project is considering all groups of customers in vulnerable situations. The number is derived from comparing the ‘Overall positive impact score’ to the ‘Overall negative impact score’. For example, projects that only have a positive impact on customers and no negative impact will have a perfect score of 10. Projects under a score of 5 have more negative impacts than benefits and should be reconsidered. 
      • Relative Impact Score – This score expresses the impact of the project out of the maximum positive or negative impacts that any project measured through this tool could have. The ‘maximum positive’ score is calculated by adding all the impacts (shown on the Impact Matrix tab) for all customer groups if a user indicates that a project would deliver all positive impacts considered in this tool. The ‘maximum negative’ score is calculated in the same way but by considering all possible negative impacts. Under this measure, a score of 10 can only be awarded if a project delivers all possible positive impacts. Similarly, a score of 0 can only be awarded if a project delivers all possible negative impacts. Generally, projects with a score above 5 have an overall benefit on customers in vulnerable situations. This score is particularly useful to compare different projects measures via this tool.
      • Positive Impact Score – This score is a measure of the positive impact that the innovation project has on all groups of customers in vulnerable situations. A score of 0 means that the project has no positive impact on customers, while a score of 10 means that the project has the highest possible positive impact that any project could achieve when assessed.
      • Negative Impact Score – This score is a measure of the negative impact that the innovation project has on all groups of customers in vulnerable situations. A score of 0 means that the project has no negative impact on customers, while a score of 10 means that the project has the highest possible negative impact that any project could achieve when assessed.
    • Customer Impact Calculator – This ‘calculator’ allows users to quickly assess the impact of the innovation project under consideration on customers affected by a specific situation of vulnerability. There are two main elements to the calculator:
      • Overall Impact Score – Unlike the other scores presented on this page, this is expressed as a statement rather than a number. The statement expresses the nature and extent of the impact of the project under consideration on a specific group of customers. The statement is based on a score that is derived from taking the simple average of the impacts that the project has on the relevant group of customers. These impacts were provided by stakeholders and are expressed between -2, the strongest possible negative impact, to +2, the strongest possible positive impact. The statement is based on the same legend included at the bottom of the Impact Matrix tab. For example, if a project has only two impacts on a customer group, and one of these is positive (+1) and the other is negative (-1), the resulting score would be 0, and the statement would be “Neutral Impact”.
      • Spider-web diagram – This shows granularity regarding the resulting 'Overall Impact Score' statement. Each point on the diagram represents a cell on the Impact Matrix where the situation of vulnerability intersects with each impact driver. In other words, the diagram offers a visual summary of the information shown on the Impact Matrix presented within the ‘Detailed Results’ tab.
    • Negative Impacts and Mitigations – This table summarises the ‘source’ of any negative impact on customers that may stem from the project being assessed by the user. The column ‘Driver of Negative Impact’ lists the aspects of the project that may deliver a negative impact on customers while the ‘High-Level Mitigation’ column includes, where possible, feedback shared by stakeholders on how negative impacts can be mitigated.
  • ‘Detailed Results’ Tab – This tab presents granular results that sit behind the summarised results included within the ‘Results Dashboard’ tab. This tab also contains a summary of (i) mitigations suggested by stakeholders for any negative impacts stemming from the project, (ii) a summary of the user inputs on the ‘Questions’ tab, along with an explanation of any answers provided, and (iii) a snapshot of the overarching results presented within the ‘Results Dashboard’ tab.                                                                                                                        

The project moved from TRL 3 to TRL 8. 

Lessons Learnt

Stakeholders’ availability – The two-week window dedicated to engaging stakeholders in the initial project plan was not sufficient. Although the engagement was extended by a week, while still delivering the project on time, ideally the engagement window would have been at least four weeks.

Methods of engagement – While we offered several methods for stakeholders to share their expert views, all stakeholders ultimately chose to fill out the survey. This was likely driven by the flexibility offered by this method. Although other methods should not be ruled out for future projects, surveys are clearly a preferred choice for many stakeholders. 

Stakeholders’ understanding of the questions posed – Some stakeholders, especially those not familiar with the energy industry, struggled to understand the questions posed in the survey. This was not driven by the wording of the question or the structure of the survey, but by the fact that they were asked to consider the impact of a project on different customer groups in a ‘typical’ scenario. These stakeholders usually believed that it would be difficult to provide views on impacts in a ‘typical’ scenario as each customer faces unique circumstances. By explaining the objective of the tool, and the role of networks the energy system (i.e., serving millions of customers across large geographical areas), stakeholders largely overcame these obstacles and provided relevant views in the survey. For future projects, it may be beneficial to prepare additional material and share this with stakeholders who are not familiar with the energy industry. This could include, for example, an overview of the energy system, the nature of a network’s relationship with customers, and how this relationship is being redefined by the energy system transition. The addition of this educational material will ensure a smoother and more effective stakeholder engagement process.

Volume of stakeholders’ questions – One of the main obstacles to stakeholder participation was the volume of questions posed in the survey. This was driven by the number of customer situations and the types of innovation projects shown within the Impact Matrix. In order to mitigate issues linked to the volume of questions on stakeholders’ participation, we introduced ‘routing’ questions that led stakeholders to only focus on questions that related to the situations of vulnerability they typically focus on. This was most effective in limiting the volume of questions posed to targeted charities (e.g., Dementia UK), but less useful for organisations that cover a wide range of situations (e.g., Citizens Advice). In future rounds of engagement, the option to randomise the questions asked to stakeholders, in addition to routing questions, could be considered to further reduce the time required to fill out the survey.